Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Abstract content storage via trait and introduce OCI content storage #146

Draft
wants to merge 8 commits into
base: main
Choose a base branch
from
Draft

Abstract content storage via trait and introduce OCI content storage #146

wants to merge 8 commits into from

Conversation

devigned
Copy link

This PR introduces the concept of a "ContentStore", which is an abstraction for storing package content. There are currently 2 implementations for a content store, local and OCI. However, it would be relatively simple to add another for a blob store like S3, Azure Blobs, R2, etc.

In this PR, the default file storage and serving directory is replaced with URIs that are served via the LocalContentStore. It should provide the same functionality as the current local file store / serving.

There is still some work to be done with the current OCI implementation to fill the image config with details more than just empty JSON. Furthermore, since the PR is using the image config media type, the artifactType must be specified, and I don't think that is yet available in oci-distribution lib. I will follow up on both of these details.

I'm opening this PR to start a discussion of implementation that will help to fill in the blanks for #87. Also, it's my first significant contribution to this repo, so I'd really appreciate guidance. Thank you!!

Testing out OCI content store support

Start a local registry

docker run -d -p 5000:5000 --restart=always --name registry registry:2

Start warg-server

Tell warg-server to use the OCI v1.1 content store. It will default to connecting to a registry insecurely at localhost:5000

david at hulkbuster in ~/code/bca/registry on oci-cs [$]
$ WARG_CONTENT_STORE=ociv1-1 cargo run -p warg-server -- --content-dir content

Publish a demo package

david at hulkbuster in ~/code/bca/registry on oci-cs [$]
$ warg publish init default:hello
published initialization of package `default:hello`

david at hulkbuster in ~/code/bca/registry on oci-cs [$]
$ warg publish release -i default:hello -v 0.1.0 demo/simple-grep-1.0.0.wasm
published version 0.1.0 of package `default:hello`

Verify the content is stored in the local registry

david at hulkbuster in ~/code/bca/registry on oci-cs [$]
$ regctl repo ls localhost:5000
WARN[0000] Changing credential host for registry         host=index.docker.io new="[https://index.docker.io/v1//refresh-token"](https://index.docker.io/v1//refresh-token%22) orig="[https://index.docker.io/v1//access-token"](https://index.docker.io/v1//access-token%22)
default/hello

david at hulkbuster in ~/code/bca/registry on oci-cs [$]
$ regctl manifest get localhost:5000/default/hello:0.1.0
WARN[0000] Changing credential host for registry         host=index.docker.io new="[https://index.docker.io/v1//refresh-token"](https://index.docker.io/v1//refresh-token%22) orig="[https://index.docker.io/v1//access-token"](https://index.docker.io/v1//access-token%22)
Name:        localhost:5000/default/hello:0.1.0
MediaType:   application/vnd.oci.image.manifest.v1+json
Digest:      sha256:ed8c29c6763e69f15204b3ed31e9a1ce5d6380a6070a67da2933d59525fcc07a
Total Size:  2.159MB

Config:
  Digest:    sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
  MediaType: application/vnd.oci.image.config.v1+json
  Size:      2B

Layers:

  Digest:    sha256:4470bd2867f49ef7119691d6e1cb9a4ecd46b79c394a7587dc78874c9382ca37
  MediaType: application/vnd.bytecodealliance.wasm.content.layer.v1+wasm
  Size:      2.159MB

@devigned
Copy link
Author

Discussed the PR with @calvinrp and @macovedj and will add an S3 compatible content store implementation to illustrate the way the client would interact with a content store which is able to send back signed URIs for fetch and upload.

@lann
Copy link
Collaborator

lann commented Jul 12, 2023

#160

@lann lann mentioned this pull request Jul 14, 2023
Merged
@jsturtevant jsturtevant force-pushed the oci-cs branch 2 times, most recently from a3c1655 to 87d6a49 Compare July 25, 2023 18:26
@devigned
fix fmt and test fails
e9d30cc 
Signed-off-by: David Justice <[email protected]>
@jsturtevant jsturtevant mentioned this pull request Aug 1, 2023
Merged
@jsturtevant jsturtevant mentioned this pull request Aug 4, 2023
Closed
@jsturtevant jsturtevant mentioned this pull request Oct 6, 2023
Closed
@vdice vdice mentioned this pull request Oct 11, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@devigned @lann